MERLOT Journal of Online Learning and Teaching 


Vol. 8, No. 3, September 2012 


Student Moderators in Asynchronous Online Discussion: 
A Question of Questions 


Daniel Zingaro 

Doctoral Student 

Ontario Institute for Studies in Education 
University of Toronto 
Toronto, ON M5S 1V6 CANADA 
daniel.zingaro@utoronto.ca 


Abstract 

Much current research exalts the benefits of having students facilitate weekly 
discussions in asynchronous online courses. This study seeks to add to what is known 
about student moderation through an analysis of the types of questions students use to 
spur each discussion. Prior experimental work has demonstrated that the types of 
questions posed by instructors influence the cognitive levels of responses, but little is 
known about the extent to which student moderators use these various question forms. 
Question types and the cognitive levels of responses in an online graduate course were 
analyzed, and it was found that students relied on a small number of question forms. In 
particular, students rarely asked questions directly related to weekly course readings, 
and did not ask any questions that made connections to previously studied course 
material. Questions that constrained student choice led to lower levels of responses 
compared to other question types. 

Keywords: asynchronous online learning, peer facilitation, peer moderation, 
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Introduction 

An increasingly popular form of distance education course is that involving the use of asynchronous 
computer-mediated communication (CMC) (Johnson & Aragon, 2003 ). Asynchronous forums typically 
use thread structures to link together related notes, allowing students to follow multiple simultaneously 
occurring discussions (Hewitt, 2005 ). Many authors highlight the benefits of threaded asynchronous CMC 
compared to synchronous CMC and face-to-face courses, including time-independent access, 
opportunities for heightened levels of peer interaction, avoidance of undesirable classroom behavior, and 
support for multiple learning styles ( Morse, 2003) . Asynchronous courses can support and embody core 
tenets of constructivist education, including participatory learning, teacher-as-collaborator, and the 
production of meaningful artifacts ( Cavana, 2009 ; Gold, 2001) . Others espouse the apparent equity of 
such courses, as such discussions tend to admit multiple perspectives and yield more even levels of 
contribution (Light, Colbourn, & Light, 1997) . Yet, while these benefits are compelling in themselves, they 
are somewhat incidental to the ultimate goal of any educational initiative: purposeful, meaningful learning. 

Unfortunately, as further described in the first subsection of the Literature Review that follows, such 
learning in online courses is anything but automatic. Various supports and pedagogical techniques are in 
use by instructors to support learning, including allowing students to facilitate weekly discussions. This 
involves giving students opportunities to shape discussions while the course instructor assumes a 
supportive, participatory role (Griffith, 2009 ; Seo, ,1007) . As described in the next section, peer facilitators 
(also referred to as peer moderators) afford many social, affective, motivational, and cognitive benefits 
compared to course instructors. One of the most important roles for peer moderators, however, is to take 
responsibility for proposing several starter questions with which to initiate a discussion. Yet, in spite of the 
importance of questioning for meaningful learning, the literature has not examined the breadth of 
questions asked by peer moderators, and their impact on participation and deep conceptual thought. This 
paper begins such an examination through an analysis of student contributions to an online course. 
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Literature Review 

Learning in Online Environments 

One of the most widely used frameworks for understanding online learning is the Community of Inquiry 
framework (Garrison, 1999 ; RQBrfce & fetiuM^iiM i At the core of this program of research is the claim 
that asynchronous learning environments can foster deep and meaningful learning in the presence of 
adequate cognitive, social, and teaching presence. Social presence refers to the feeling that others are 
"actually there" in the environment, whereas teaching presence reflects the instructional, facilitative, and 
organizational roles of the instructor. Cognitive presence is defined as "the extent to which the 
participants in any particular configuration of a community of inquiry are able to construct meaning 
through sustained communication," and is measured by four forms of discourse: triggering events, 
exploration, integration, and resolution ( Rourke & Kanuka, 2009 , p. 21). The last two of these categories 
represent deep, high-level learning, and so the question is: can and does this type of learning occur in 
asynchronous courses? 

Rourke and Kanuka (2009) synthesize several references suggesting that the vast majority of student 
posts fall in the lowest levels of cognition, and that only 1-18% of posts can be considered "resolution" 
posts. Slightly more promising results were offered by Meyer (2004) , who found 52% of notes in two 
doctoral education courses to be at the integrate-resolve levels, and 20% of posts to be at the resolution 
level. Schrire (2006) noted that integration and resolution may be increased in synergistic threads 
compared to instructor-centered threads, the latter of which are dominated by instructor posts and 
questions. She found, for example, that 24% of notes in instructor-centered threads contained evidence 
of integration and resolution, compared to 53% in synergistic threads. Others have used Bloom's 
taxonomy to determine the extent to which higher level cognition occurs, often finding quite sobering 
results. The approach involves treating the top two levels of Bloom's (1956) taxonomy - synthesis and 
evaluation - as reflecting higher order thinking, and the other four levels as low- or medium-level thinking. 
One study found that only 1 5% of student posts were at the highest two Bloom's taxonomy levels ( Ertmer, 
Sadaf, & Ertmer, 2011) ; others found that student posts are highly concentrated in the analyze-apply 
midrange ( Bradley, Thom, Hayes, & Hay, 2008 ; Christopher, Thomas, & Tallent-Runnels, 2004) . 

While the focus in this paper is on the relationship between question type and the cognitive level of 
discussion, the research cites many other factors that affect discussion quality. Online discussions can 
take many forms (Hammond, 2005 ), and the learning activities chosen by the instructor can impact the 
level of critical thinking engaged in by students (Richardson & Ice, 2010) . Garrison and Arbaugh (2007) 
suggest that the difficulty in moving through the inquiry process can be traced to aspects of teaching 
presence. The types of questions asked by the instructor are known to influence subsequent interaction, 
and some types (e.g., those that have practical applications) foster increased cycling through the entire 
inquiry process. Both the direct-instruction component (Garrison & Arbaugh, 2007 ) and discourse- 
facilitation component (Gorsky & Blau, 2009) of teaching presence are critical for moving students to 
resolution. Teaching presence is also critical for perceived learning, student satisfaction, and online 
community ( Gorsky & Blau, 2009) . 

Student Moderators 

Interaction and participation are two key constructs thought to enhance learning in asynchronous 
courses. Hammond (2005) describes this assumption as arising from teachers' alignment with social 
constructivist and collaborative learning principles. Engagement in online learning requires student- 
student and student-teacher interaction, and so teachers should strive to form supportive online learning 
communities (Rovai, 2007 ). Yet, such communities are difficult to build in the face of certain recurring 
challenges to online learning: an overwhelming quantity of messages to read, increased likelihood of 
misunderstandings, and reduced student motivation. Both Rovai as well as Collins and Berge (1996) note 
that the key to overcoming such challenges lies in skillful instructor facilitation. This facilitation/moderation 
role is multifaceted, containing such aspects as being supportive, weaving and focusing topics, and 
offering coaching and leadership (Tagg, 1994 ). Instructors should seek to develop social presence, avoid 
becoming the center of discussions, and emphasize student-student interactions, all while encouraging 
students and indicating that their posts are being read. When instructors become central, the otherwise 
communal forum can degenerate into a question-and-answer session ( Mazzolini & Maddison, 2003 ). 

To increase student-student interaction, researchers have begun investigating the viability of having 
teachers assign or students self-select moderator roles. As moderators, students can play four 
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interdependent roles (Wang, 2008 ): intellectual/pedagogical, social, managerial, and technical. Novice 
peer moderators tend to ascribe most importance to the social (e.g., encouraging participation, inviting 
responses) and intellectual (e.g., summarizing) roles. Indeed, this pattern also appears to be reflected in 
teacher-supplied guidelines for peer moderators, of which Griffith (2009) offers a representative example. 
In his study, the moderator was required to (p. 38): 

• provide guidance on the reading and research for the week, identifying the particular focus to be 
pursued; 

• pose questions to guide the on-line discussion; 

• initiate and stimulate discussion for the topic under consideration; and 

• guide discussion by (a) logging on daily to integrate and advance the discussion, (b) ensuring 
that all questions were discussed and (c) encouraging each participant to arrive at some closure. 

In some of the earliest work on the subject, Tagg (1994) argued that student moderation sets up a 
context within which students and teachers can complement one another's strengths. In his study of 
students in graduate psychology courses, each week one student signed up for the role of topic leader 
(submitting the initial contribution) and another signed up to be topic reviewer (submitting a final, 
synthesis posting). It was found that such student involvement promoted increased cohesion and 
structure to the discussions. Furthermore, interaction and participation was increased by virtue of having 
students publicly commit to their moderator roles. Hara, Bonk, and Angeli (2000) corroborate this finding: 
forums headed by students acting as "starters" yield fewer isolated and scattered discussions. Tagg and 
others (e.g., Rourljl & Aiiderslri, 2002 ) invoke an empowerment argument to explain that instructor- 
posted opener/starter notes tend not to be as effective as starters posted by students: students may 
hesitate to respond to the instructor, whom they perceive as being more knowledgeable and competent. 
At the same time, however, the instructor remains critical for providing recognition and requesting 
clarifications and elaborations. In another early study ( Murphy et al., 1996) , increases in participation 
similar to those reported by Tagg were obtained when the instructor selected the weekly moderators, 
rather than having students self-select. 

To foster this increased participation, it is not sufficient for peer moderators to post frequently themselves 
(Chan, Hew, & Cheung, 2009 ). What is it about moderators, then, that does tend to enhance 
participation? Hew and Cheung (2008) sought to answer this question through a study of thread structure 
in blended courses. They argued that deep threads (operationalized as having a depth of at least six) 
implied that significant participation had occurred, and therefore proceeded to examine such threads for 
salient moderator tactics. Seven facilitator strategies were found: disclosing personal opinions, 
questioning, showing appreciation, establishing ground rules, suggesting new directions, inviting to 
participate, and summarizing. Of these, questioning occurred most frequently, in a combination of asking 
for clarification and asking for students' opinions. And, while questioning tends to promote thread growth, 
summarizing tends to cut threads short (Chan et al., 2009) . 

Compared to face-to-face courses, asynchronous courses permit students to assume a more central 
teaching-presence role ( Heckman & Annabi, 2005 ), especially when those students are assigned to 
moderate weekly discussions ( Rourke & Anderson, 2002) . Rourke and Anderson argue that addressing 
the constituents of teaching presence can be daunting for a single teacher, and that student moderation 
may help distribute and balance such responsibilities. These authors describe an asynchronous graduate 
course in which the instructor modeled moderation for the first five weeks, and then teams of four 
students moderated each of the remaining weekly discussions. Content analysis of the course transcripts 
showed that peer moderators exceeded the instructor on percentage of instructional design and 
facilitation posts, and that direct instruction was comparable between peer moderators and the instructor. 
Questionnaire data and interview analysis confirmed that the student moderators performed comparably 
to the instructor on all three teaching roles. 

It is important here to add that the instructor's role remains critical in a course where peer moderators are 
used. Rourke and Anderson (2002) note that the instructor can work to fill teaching-related gaps left by 
more pedagogically novice student moderators. Furthermore, instructor-controlled elements of online 
structure can impact the effectiveness of peer-moderator activities. Gilbert and Dabbagh (2005) report a 
multiple case study of four increasingly structured iterations of a blended graduate course, wherein pairs 
of students moderated weekly discussions. In the first offering of the course, peer moderators were 
simply told to post at least two starter questions, facilitate the discussion, and synthesize the discussion in 
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a follow-up face-to-face meeting. In the second offering, moderators were provided with a document 
outlining their specific roles and a list of criteria on which they would be evaluated by the instructor. These 
resources increased the percentage of posts that were written by the moderators, including posts that 
responded to other students, clarified material, and asked follow-up questions. This increased moderator 
involvement led to greater interaction among students and heightened "meaningful discourse" as 
measured through inference, analysis, synthesis, and evaluation. Finally, it is important for instructors to 
model and motivate the role of peer moderator prior to having students assume facilitation roles ( Baran & 
Correi&:#QQ9T Baran and Correia describe a case study in which the instructor modeled online 
facilitation for three weeks, stressed the importance of facilitation skills to the soon-to-be teachers in the 
course, provided guidelines, and encouraged the exploration of meaningful and novel facilitation styles. 
Student facilitators readily responded to this motivating, safe venue for leading their groups and 
encouraging meaningful interaction. As an example, one student chose to lead through a highly 
structured style that kept others focused on current learning goals in close connection to the week's 
readings and their own teaching practice. Another facilitator asked students to recount personal teaching- 
related goals and wishes so as to create an atmosphere of trust and personal disclosure. In each case, all 
students were active and enthusiastic participants, taking ownership of the discussion and others' 
learning. It is unlikely that such varying facilitation styles, motives, and learner engagement would follow 
from an instructor not attuned to the nuances of giving students power and "letting go" (Gunawardena, 
1992) . 

Online Questioning 

Decades of research in face-to-face settings has focused on two broad classes of questions: lower 
cognitive and higher cognitive, mapping, respectively, to the knowledge and comprehension and the 
application, analysis, synthesis, and evaluation levels of Bloom's taxonomy (Gall, 1970 ; Redfield & 
Rousseau, 1981) . Lower cognitive questions ask students to recall or recognize information that has 
previously been presented, whereas high-cognitive questions demand evidence-based, information- 
informed reasoning processes. A meta-analysis of 14 school-based studies found that average students 
treated with high-cognitive questions reached the 77th percentile of achievement, compared to the 
expected 50th percentile ( Redfield & Rousseau, 1981) . A follow-up research synthesis tempers this 
finding, but concludes that high-cognitive questions do have a small effect overall (Samson, Strykowski, 
Weinstein, & Walberg, 1987) . Gall (1984) explains that while it is a common belief that the cognitive level 
of the question has a strong influence on the cognitive level of the responses, this is not always borne out 
by the literature. 

Relatively few studies, however, have examined the effects of questions in the online setting, and it is 
dangerous to assume that what is true in face-to-face settings will be true online. As one example shows, 
students often have limited opportunities to ask questions in face-to-face settings, whereas Blanchette 
(2001) found that 77% of questions in an online graduate course were written by students. Blanchette 
further classified questions and their responses using three different frames of analysis: linguistic, 
pedagogic, and cognitive. While these frames each contributed to the depth of the present study, the 
main interest here is in the cognitive frame. The cognitive classification was based on a coding system 
developed by Gallagher and Aschner (1963) , who grouped questions as routine, cognitive-memory, 
convergent, divergent, or evaluative. Blanchette found that evaluative questions were used most 
frequently by the students and teacher, that divergent questions were used very infrequently, and, overall, 
that most questions written by the students and teacher were at the high cognitive levels. In general, the 
cognitive level of responses to an instructor-posed question matched the cognitive level of that question. 

Using a typology of question types from Andrews (1980) , several studies have found that the type of 
question influences the quantity and quality of subsequent interaction. Bradley et al. (2008) classified 
questions into the following categories of Andrews': 

• Direct link: ask students to interpret an aspect of a course reading; 

• Course link: ask students to integrate specific course knowledge with the topic of a reading; 

• Brainstorm: ask students to generate any and all relevant ideas for or solutions to an issue; 

• Limited focal: present an issue and several alternatives and ask students to justify a position; 

• Open focal: ask for student opinion on an issue without providing a list of alternatives; 

• Application: present a scenario and ask students to respond using information from a reading. 
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Bradley et al. balanced the use of these prompt types over three sections of an online undergraduate 
course on child development. Limited-focal and direct-link questions generated the most words, whereas 
limited-focal and open-focal questions generated the most complete answers. Application and course-link 
questions were least effective in terms of word count and answer completion. Using a coding scheme 
based on Bloom's taxonomy (Gilbert & Dabbagh, 2005 ), course-link and brainstorm questions were found 
to be best for higher order thinking, though thinking overall was concentrated at a comprehension level. 

A similar study was conducted by Ertmer et al. (2011) , who purposively sampled 10 graduate courses for 
instructor prompts that covered the various Bloom's taxonomy levels and each of Andrews' question 
types. They found that questions at high Bloom's taxonomy levels facilitated responses at those high 
levels, though still with one-third of responses at the comprehension level. In terms of Andrews' types, 
lower divergent questions (those that ask students to generate conclusions and generalizations from 
data) were most promising in yielding responses at the medium and high levels of Bloom's taxonomy. In 
stark contrast to the results of Bradley et al. (2008) , brainstorm questions led to a majority of responses in 
the low Bloom's taxonomy levels while yielding the highest number of posts per student and response 
threads. 

Such findings - that the level of question influences the level of response - are mirrored by those studies 
using Garrison, Cleveland-lnnes, and Koole's (2006) cognitive-presence indicators ( McLoughlin & 
Mynard, 2009) . Meyer (2004) found in an analysis of 17 doctoral-level online discussions that almost half 
(40%) of the resolution posts were concentrated in five discussions whose leading questions explicitly 
asked students to solve a problem. 

Investigations into the questions written by student moderators are rarer than the abovementioned 
studies focusing on teacher moderators. One study (Christopher et al., 2004) examined the relationship 
between questions and responses posted by 10 students in a blended graduate gifted-education course. 
Each student posted one prompt in one of the 10 discussion weeks. The results indicated that most of the 
responses were at the levels of application and analysis, though four students consistently synthesized 
and evaluated. Further, in contrast to the teacher-moderated studies cited earlier, there was no pattern 
linking the Bloom's taxonomy level of the prompt and the average cognitive level of its responses. Yet, in 
a content-analytic study using a well-traveled analysis tool ( Hara et al., 2000 ), it was found that the 
questioning activities of "starters" held considerable sway in shaping incident discussion. For example, 
when the majority of the starters' questions were "inference questions," the most frequently used 
cognitive skill during the discussion was inference. To be sure, results from face-to-face studies of 
student-generated questions suggest that the inconsistency between these two studies is no coincidence: 
some such studies indicate that student-generated questions are positively correlated with achievement, 
while others do not ( Waugh, 1996) . 

Here, it is suggested that categorizing questions along Andrews' categories may bring some clarity to 
these disparate results. After all, Andrews' categories have been found to be extremely powerful 
predictors in several instructor-focused studies ( Bradley et al., 2008 ; Ertmer et al., 20.11 ). In addition, 
should such a categorization prove fruitful in understanding student questioning patterns, it would be able 
to serve as a ready-made typology for teaching and encouraging students to preferentially use certain 
types of questions (Choi, Land, & Turgeon, 2005 ). 

Method 

Context 

The course studied here is a fully online graduate education course that took place in Fall 201 1 at a large 
Canadian research university. The course used an online learning environment that supported both 
asynchronous and synchronous communication. The former was implemented through weekly discussion 
forums and is the interest of this paper; the latter took place in private chat spaces and is not analyzed 
here. The course concerned various topics related to the educational use of asynchronous and 
synchronous CMC, including its history, the role of the teacher, student factors, and Web 2.0 
technologies. There were 11 modules, each corresponding to one week, in which students discussed 
instructor-assigned readings. The class was sufficiently small (13 students) that the instructor decided to 
have all students discuss in a single large group rather than splitting them into smaller weekly discussion 
groups. Each week, one or two students acted as moderators. The moderators carried out roles in 
accordance with those specified by the literature (Griffith, 2009 ): they collaborated in advance to develop 
guiding questions for the week, facilitated discussion throughout the week, and finally offered a summary 
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of the week's issues. The instructor provided moderators with literature and best-practice strategies for 
focusing, maintaining, and extending discussions. Each student acted as moderator once during the 
course, and such moderation accounted for 20% of their course grade. Information relating to students' 
prior moderation experience was not collected. Students' contributions to the weekly discussions (not 
including the moderator roles just described) were worth 30% of their grade. 

For the first three weeks of the course, the instructor served as moderator. She posted starter questions, 
encouraged students to express their viewpoints, and otherwise modeled the moderator role for the 
students. The remaining eight weeks were moderated by students, with the instructor participating 
alongside the other students in the forum. 

Instruments 

The present study made use of descriptive quantitative content analysis, the goals of which are to be 
systematic and objective. By systematic it is meant that the coding categories are determined a priori; by 
objective it is meant that classification is subject to inter-rater reliability checks and that raters sufficiently 
agree on categorizations ( Rourke & Anderson, 2002 ). Rourke, Anderson, Garrison, and Archer (2001) 
note that there is a lack of replication studies in the online learning literature, which poses a threat to the 
establishment of widely applicable and validated instruments: while many instruments are being created, 
they are rarely being used outside the institution in which they are developed. Moreover, since such 
instruments often infer cognitive processes from the text resulting from such processes, validity in these 
schemes is anything but assured. Rourke and Anderson (2004) suggest an unfortunate catch-22 in the 
use of quantitative data analysis in online learning research: creating new coding schemes ought to be a 
meticulous process of theoretical and empirical validation, whereas many existing instruments have been 
developed in the absence of such rigor. Nevertheless, confidence in existing schemes can be enhanced 
through their use in new studies and contexts that attest to the robustness and generalizability of their 
categories and indicators. For this reason, two instruments that had been used in previous related studies 
were adopted: to classify questions, the Andrews-based typology of Bradley et al. (2008) was used as a 
starting point; to measure higher order thinking in discussions, the Bloom-based coding scheme of 
Bradley et al., which was adapted from Gilbert and Dabbagh (2005) , was employed. 

These coding schemes have been followed in the present study with the choice of the message posting 
as the unit of analysis. Rourke et al. (2001 ) describe the tradeoff between unitizing syntactically (e.g., at 
the message or paragraph level) and semantically (e.g., thematic unit) in terms of coder agreement and 
construct representativeness. For example, units such as sentences and paragraphs may not be easily 
separable in online learning transcripts when participants use ellipses or extra blank lines, and tend to 
yield an unruly number of units. In contrast, messages are objectively identifiable, are far fewer in number 
than smaller syntactic units, and structurally correspond to the intentions of their authors. 

Reliability 

There are two broad classes of coding in quantitative content analysis: manifest coding and latent coding 
(Lombard, Snyder-Duch, & Bracken, 2002) . The former involves coding features of communication that 
are readily observable, such as word counts or use of pre-specified emoticons; the latter requires 
subjective interpretations to infer those processes responsible for manifest content. Rourke and Anderson 
(2002) explain that inferring quantitative aspects of online postings, as has been done here, is an 
example of the latent level. Such coding in particular requires that inter-rater agreement be both 
established for purposes of intersubjective judgments and carefully described for purposes of 
communicating with other researchers. 

Among the many available measures of inter-rater agreement, percent agreement, Cohen's kappa, and 
Krippendorffs alpha are frequently used (Lombard et al., 2002 ). Literature deems that percent 
agreement, reported on its own, is inappropriate because it does not take into account agreements that 
occur by chance. The interpretation of Cohen's kappa depends on the number of coding categories and 
the distribution of the data, so it can be difficult to make comparisons across studies ( Rourke et al., 2001) . 
In contrast, Krippendorffs alpha is highly recommended H rtebarai et al.. 2111 1 as it is a conservative, 
chance-corrected measure that supports all levels of data and any number of coders. Krippendorffs alpha 
is therefore reported in coding the responses to moderator questions. 

Garrison et al. (2006) describe a negotiated approach to coding that involves researchers coding 
transcripts and then discussing those codes to arrive at maximal consensus. In contrast to traditional 
multiple-coder analysis, negotiated coding involves all coders coding and agreeing on the same analysis 
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units. While certainly more time consuming, negotiated coding may be particularly effective in initial, 
exploratory research (Garrison et al., 2006 ). A decision was made to adopt this approach for coding the 
moderator questions in the present study. Such questions were few compared to the number of response 
notes, and therefore negotiating a code on all questions was thought to be manageable in terms of time 
and resources. In addition, and to the extent feasible, it was considered important in coding these 
questions to "get them right," since the categorization of a question would determine the category into 
which all of its responses would be placed. 

The present author and a research colleague engaged in this process of negotiated coding. They began 
by creating a codebook containing descriptions and examples of the codes. They then trained themselves 
on the coding instrument using questions from the prior (Winter 201 1 semester) offering of the course. 
Next, they independently coded each question, and finally met to negotiate coding on all questions. The 
results of this coding, along with meta-information about the coding process itself, are described further in 
the next section. 

Results 

Moderator Questions 

As noted above, the author and his colleague began with the typology of questions used by Bradley et al. 
(2008) (see the "Online Questioning" subsection in the Literature Review for their six categories). 
However, they very quickly noticed that many starter notes in fact contained several often-unrelated 
questions that could not be categorized into one of these six types. For example, in a week on web 
accessibility, the student moderators posed the following starter question: 

[After hyperlinking to and briefly summarizing an article asserting that web accessibility is a right] 
"How does such an assertion hold up to your educational experiences using technology in the 
classroom ? Is it possible that such technologies are sometimes treated more like a privilege than 
a right?" 

The first of these two questions asks students to relate directly to a reading-based statement, and would 
therefore be classified as direct link (DL). The second appears to be a limited-focal (LF) question, asking 
students to decide whether web accessibility in practice is a privilege or a right. To account for these 
multifaceted questions, the shotgun (SG) category from Ertmer et al. (2011) was added. The seven codes 
are presented in Table 1. 


Table 1. Types of questions 


Code 

Shorthand 

Description 

Direct link 

DL 

Refers to a specific aspect of an article (e.g., a quotation) and asks 
students for interpretation or analysis. The direct link to the article can 
be found in the question itself or be a requirement in the response. 

Course link 

CL 

Asks students to integrate specific course knowledge with the topic of 
the article. 

Brainstorm 

BS 

Asks students to generate any and all relevant ideas or solutions to an 
issue. 

Limited focal 

LF 

Presents an issue with several (e.g., two to four) alternatives and asks 
students to take a position and justify it. 

Open focal 

OF 

Asks for student opinion on an issue without providing a list of 
alternatives. 

Application 

AP 

Presents a scenario and asks students to respond using information 
from a reading. 

Shotgun 

SG 

Presents multiple questions that may contain two or more content areas. 


The negotiated coding process yielded useful information about the coding scheme. For example, the 
coders repeatedly had difficulty agreeing on whether a question was open focal (OF) or brainstorm (BS). 
In fact, the only question on which they could not ultimately agree was of the following form (only the 
question stem has been included so as to comply with a student's consent requests): "do you see any 
advantages to [new technology]?" The present author classified this as a BS question, arguing that, in 
meaning if not in syntax, the question was asking for a "braindump" of perceived advantages. The second 
coder, on the other hand, argued that this was an OF question that was asking for an opinion: are there 
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any advantages of this new technology? Ultimately, they could not come to an agreement, and arbitrarily 
classified the question as BS. 

Table 2 indicates the number of instructor (first three weeks) and student (remaining eight weeks) 
questions. Most weeks contained three or four questions; the final week, however, was a strong outlier 
containing 13 questions (nine BS, one DL, one LF, two SG). Of the 40 total student questions, the vast 
majority (29) were BS or SG. BS questions often asked students to produce descriptions of personal 
experiences relevant to the topic at hand, or generate many examples of a phenomenon. For example, 
one of the questions in Week 4 (on the topic of social networking) asked, "What examples do you know of 
instructors using SNS [social networking sites] in their teaching?" In Week 6, student moderators asked, 
"After watching these two videos [external hyperlinks to two new video resources], describe the radical 
changes you would make to teaching in CMCs. Why would you make these changes?" 

SG questions often contained components spanning question types and topic areas. For example, again 
in Week 4, student moderators asked: 

"Do you have SNSs that you are in favour of (If you don't please pick one from boyd and Elison 
(2008) [a course reading for the week])? What would be potential benefits for using it as teaching 
tool? What would be your issues/concerns using it as teaching tool?" 

The first subquestion here is LF, asking students to choose among the available SNSs; the latter two 
questions are highly related BS questions. 

Students did not ask any application (AP) or course-link (CL) questions. Further examples of student 
questions falling into each of the other five categories are provided in Appendix A. 

The instructor asked a total of nine questions, which were also categorized using the negotiated coding 
process. Similar to what was described for the students' questions, the majority of instructor questions 
(five questions) were again BS or SG, with none of them classified as AP or CL. 

Table 2. Question types produced by the instructor and students 


Written By 

Question Type 

BS 

DL 

LF 

OF 

SG 

Instructor 

3 

1 

0 

0 

5 

Student 

17 

3 

3 

5 

12 


Cognitive Level of Responses 

Given that the interest of the present study is primarily in the responses to student (rather than instructor) 
questions, the first two weeks of discussion were used as the reliability sample. The coders began by 
training themselves on the instrument using the first question from Week 1, then independently coded 
responses for subsequent questions. It took three rounds of coding responses to single questions to 
reach a Krippendorff's reliability of .7. (The ordinal alpha was used, so as to incorporate the distance 
between coders, in addition to exact agreement.) After this, the present author coded 80% of the notes 
produced in the eight weeks of student moderation, and the secondary coder coded the remaining 20%. 
Both coders had been students in prior offerings of the course, and had previous research experience 
with Bloom's taxonomy. 

Table 3 contains the instrument used for coding the responses, based on that of Bradley et al. (2008) . For 
each question type, the percentage of each participant's notes that fell within the low (no score, reading 
citation, or content clarification), medium (prior knowledge, real-world example, or abstract example), and 
high (making inferences) levels of Bloom's taxonomy were calculated. Then, for each question type, the 
average percentage of low notes was obtained as the average of the students' low percentages for that 
question type. This was repeated for the medium and high percentages. In this way, each student's 
contributions, no matter how many, were equally weighted in the calculation of the per-question-type 
percentages. Figure 1 presents the results of this analysis. 
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Table 3. Cognitive response categories 


Code 

Bloom's Level 

Description 

No score 

N/A 

Student attempted submission, but it cannot be coded as a 
result of being too off topic or incorrect. 

Reading citation 

1 

Student only cited the reading using mostly direct quotations 
when justifying his/her answer. 

Content clarification 

2 

Student stated personal interpretation of article content, such 
as paraphrasing ideas in his/her own words. 

Prior knowledge 

2 

Student used prior knowledge or referred to new outside 
resources when justifying his/her answer. 

Real-world example 

3 

Student applied a personal experience or scenario to justify 
his/her answer. 

Abstract example 

3 

Student applied an analogy, metaphor, or philosophical 
interpretation to justify his/her answer. 

Making inferences 

4, 5, 6 

Student's answer reflected analysis, synthesis, or evaluation, 
and he/she made broader connections to society or culture and 
created new ideas in justifying his/her answer. 


o 



Question Type 

Figure 1 . Frequencies of responses by question type 

LF questions yielded substantially lower inference percentages than other question types. The three LF 
questions asked students to state whether they preferred synchronous or asynchronous communication, 
choose the most important factor from among five influences on their participation in a community of 
practice, and argue for the maturity or immaturity of a new technology. In each case, the restricted range 
appears to have directed discourse toward stating choices, rather than comparing alternatives, making 
value judgments, or relating impacts of choices to other knowledge. A brief consideration of the 
responses to the first LF question allows this to be illustrated. The initial response made comparisons 
between the use of asynchronous/synchronous tools, arguing that the "best" medium might differ 
depending on whether one is a student or a teacher. This inferential response was followed by several 
notes expressing approval for this nuanced approach. Others brainstormed advantages and 
disadvantages of one of the media, or provided personal examples of why they favored one mode over 
the other. Of the 10 responses to the question contributed by a total of eight participants, only two notes 
were coded as involving making inferences. In fact, across all three LF questions, the two students that 
wrote these notes were the only students to reach any inference. 

SG questions yielded inference percentages as high as 44% and as low as 0%, though the latter appears 
to be an outlier. The 0% SG question discussed open educational resources (OER), and asked many 
questions: "Have you re-used/re-mixed OER? What were the advantages and challenges? Do you have 
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an OER you found useful? How can OER help public good be met?" The 0% inference resulted from 
students focusing only on sharing their favorite OERs; one cannot help but wonder to what extent 
students would have reached the inference level had they chosen to tackle other subquestions. Inference 
in BS questions peaked at 40%, and, overall, BS questions gave rise to a similar mean inference level as 
SG (Wilcoxon signed-rank test, W = 0.58, p = .63). 

Discussion 

When looking at the types of questions produced by student moderators, a heavy reliance is found on two 
types of questions: SG and BS. Three comments are made relating to this finding. 

First, it can be hypothesized that this paucity of variety in question types may be partly responsible for the 
preponderance of lower and medium-order thinking in the present course. Bradley et al. (2008) found that 
CL and DL questions yielded the highest levels of thinking, yet these questions were near absent in the 
course that was analyzed. And, while BS questions were lauded in the Bradley et al. study and on par 
with other question types in the present study, they were found ineffective in Ertmer et al. (2011) . 
Similarly, both the findings of Ertmer et al. and those of the present study point to the fact that SG 
questions promote thinking at the midrange of Bloom's taxonomy. In summary, there is no solid evidence 
that the types of questions asked by the students in the present course led to higher order thinking. Of 
course, there is also no direct evidence that other types of questions would have led to higher levels of 
thinking, but this is hypothesized to be the case based on the literature cited here. 

Second, there is reason to believe that CL questions in particular might play a role in the ability of the 
course to promote knowledge advancement. Scardamalia (2000) argues that students can engage in 
knowledge building in the presence of four technology-supported processes. One such process is the 
continual improvement of ideas: acknowledging that knowledge is not fixed, but that it can and should be 
continually refined. Community of inquiry researchers argue similarly: that integration of ideas is a desired 
cognitive goal, and that this requires students to synthesize what has been said in order to move ideas 
outside of their initial contexts (Garrison, 1999) . It is suggested that if students in an online course are to 
do this, they must repeatedly reconsider knowledge built in earlier weeks, and that one way to promote 
this behavior is through CL questions that require old ideas to remain pliable as students work to connect 
and integrate the body of course knowledge. 

Third, it is unclear to what extent students' online question preferences in the present course were 
reflective of instructor modeling. The instructor, by virtue of modeling the first three weeks, likely played a 
role in the types of questions that were subsequently asked by students, particularly for those students 
who were new to online learning. To the extent that students lack experience in asking questions, it is 
suggested that the instructor's questions can be powerful forces in shaping question-asking patterns. If 
the instructor uses no CL questions, as was the case in the present study, students might not consider 
CL as an available option, might believe that CL questions are not valued, or might not understand the 
possible educational potential of CL questions. While it is believed the instructor should consciously 
model several question types, it may not be necessary to model all such types. Instead, students could 
be supplied with a list of types of questions that they might consider using, and be encouraged to 
experiment with a variety of those types in their weekly moderation activities. In fact, there is a particular 
difficulty here in modeling CL questions, since by definition they require the existence of prior course 
material. If the instructor models during the first few weeks, it will be challenging to make meaningful links 
to earlier material. Perhaps the instructor would do well to moderate in Week 1, and then again in a 
slightly later week (e.g., Week 5), in order to demonstrate a wider variety of question types. 

Limitations of the Study 

There are three main areas in which limitations of the present study exist. First, it has been hypothesized 
throughout this paper that the type of question is a determinant of the type of interaction. However, other 
aspects of student moderation that might interact with question type have not been considered. In 
particular, holding question type constant, it seems reasonable to assume that individual moderator styles 
will work to influence the type and level of responses. In studies where questions are all created by the 
instructor who moderates each week, this is a non-issue. Unfortunately, with different students acting as 
moderator each week, it is difficult to decouple the question itself from the student asking the question. 

Second, no distinction has been made between direct responses to a question and indirect responses 
(i.e., responses to responses). The number of direct responses was small, leading to consideration of the 
entire response set as the data to be analyzed for each question. Yet, other researchers (e.g., Bradley et 
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al„ 2008) have found differences between initial responses to questions and subsequent replies to those 
responses. 

Third, the limited number of question types, and the small number of questions of most other types, 
precludes the making of strong judgments on the most useful questions. In future work, attempts will be 
made to help students broaden their use of question types, and then carry out a more complete analysis 
of all question types that are of interest. 

Conclusion 

Compared to other studies that vary the types of instructor-posed questions or look across courses for 
variability in questions, it was found in the present study that student moderators in the online course 
used a limited variety of question types. In particular, there was no evidence of questions being asked 
that pertained to content covered earlier in the course, and little evidence of questions relating directly to 
the current readings. Students asked mostly brainstorm questions or questions that tapped multiple areas 
of interest using different question forms. To benefit from other types of questions known to spur higher 
level discussion, future research should seek to encourage the use of question variety among student 
moderators. The author has hypothesized that teacher modeling and/or making available question-related 
resources would help students understand the value of varying question form. In general, the analysis of 
responses reported in this paper accords with other research that finds large portions of online discourse 
to be focused on the lower levels of Bloom's taxonomy. Whether students can successfully use direct-link 
and course-link questions and whether such questions yield higher level responses when they do so are 
important open questions. 
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Appendix A: Question-Type Examples 


Code 

Example 

DL 

"Coiro (2003) asserts that we "can no longer allow the fears of adults to dictate or confine the 
literacy needs and desires of the young readers and writers of our future. " With the dearth of 
resources available to students apart from the instructor, is it not possible for students to still 
have their needs met even if their instructor is fearful of using digital texts and tools? Is there 
any danger in this?" 

BS 

"This article [hyperlink to external article] describes a neomillennial learning style arising out of 
new technologies (p. 10). On the other hand, another study (Margaryan, Littlejohn, & Vojt, 2011) 
found that "Students in our sample appear to favor conventional, passive, and linear forms of 
learning and teaching. Indeed, their expectations of integration of digital technologies in 
teaching focus around the use of established tools within conventional pedagogies" (p. 439). 
What has been your experience in your own practice, as teacher and/or student, of students' 
learning styles? How do you see technology making a difference?" 

LF 

"In pages 503-506, Sherblom (2010) identifies some influences on instruction and learning in 
the CMC classroom. In your initial experience with the Community of Practice you selected for 
this course, which of the following influences most hindered/promoted your participation in the 
CoP and why? 

1 ) Interpersonal uncertainty reduction; 

2) Social presence, anxiety, and apprehension; 

3) Social interaction, experience and training; 

4) Social identity and anonymity; 

5) Identity in social relationship. " 

OF 

"Brunvand and Abadeh's article also touches on the fact that several Web 2.0 resources are 
free for students. Unmentioned, however, is the fact that online resources (such as Google, 
YouTube, and Facebook) are often provided by private corporations, undoubtedly driven by 
profit incentives. Is there a concern of commercializing our students by embedding so much of 
their learning opportunities into private corporations?" 

SG 

"This article [hyperlink to external article] considers how "Second Life" or SL, an online virtual 
world, is a CoP [Community of Practice] environment which can have applications in education. 
It might be helpful to watch this virtual campus introductory video [link to external video]. 

In reference to Gorini et at. (2008) [external article], the article describes an example from the 
telehealth sector: "... compared with conventional telehealth applications such as emails, chat, 
and videoconferences, the interaction between real and 3-D virtual worlds may convey greater 
feelings of presence, facilitate the clinical communication process, positively influence group 
processes and cohesiveness in group-based therapies, and foster high levels of interpersonal 
trust between therapists and patients" (p. 2). 

In the context of education, can you imagine virtual worlds building group trust and facilitating 
positive interactions? Could they be more effective at it than environments such as Pepper 
[institutional environment in which the course was run]? Is there anything in your own 
experience which could affirm or deny this? How might this course be different if it were 
conducted in Second Life?" 
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